Skip to content

fix: Lambda GPU image security hardening#5733

Merged
junpuf merged 24 commits intomainfrom
patch-lambda-os-cves
Mar 13, 2026
Merged

fix: Lambda GPU image security hardening#5733
junpuf merged 24 commits intomainfrom
patch-lambda-os-cves

Conversation

@junpuf
Copy link
Contributor

@junpuf junpuf commented Mar 11, 2026

Description

Comprehensive security hardening for Lambda GPU runtime images, reducing the ECR scan allowlist from 35 → 4 entries.

Changes

OS patching

  • Add dnf upgrade -y --security --releasever latest to all 3 runtime stages to pick up AL2023 security fixes blocked by the NVIDIA base image's pinned release version

Python dependency hardening

  • Replace full COPY --from=lambda-python /var/lang /var/lang with selective copy of bin, include, lib only, then rm -rf site-packages
  • Install all Python dependencies via uv so we own the full dependency tree (no Lambda-bundled boto3/botocore/urllib3/etc.)
  • Explicit pins: awslambdaric==3.1.1, boto3==1.40.4, setuptools==82.0.1, pip==26.0.1, urllib3==2.6.3, etc.
  • Remove ensurepip/_bundled — CPython-bundled pip wheel containing old vendored setuptools/urllib3; not needed at runtime
  • Retain pip in runtime images for customer extensibility (consistent with existing Lambda runtime images)

CVEs resolved (31 removed from allowlist)

Category Count Fix
AL2023 OS CVEs ~21 dnf upgrade --releasever latest
urllib3 1.26.x (ensurepip bundled wheel) 1 Remove ensurepip/_bundled
jaraco.context (setuptools vendor) 1 Bump setuptools to 82.0.1
Python tarfile stdlib CVEs 5 fixAvailable=NO — scan filter correctly ignores unfixable CVEs
GnuPG tpm2daemon 1 fixAvailable=NO — scan filter ignores
Go CVEs (misattributed to NVIDIA) 2 Actually in aws-lambda-rie; corrected allowlist reason

Remaining allowlist (4 entries — irreducible)

CVE Reason review_by
CVE-2024-6345, CVE-2025-47273 setuptools 38.5.0 vendored inside pip/_vendor/pkg_resources — structural limitation of pip, present in all pip versions; vendored code is unreachable by user code and never invoked at Lambda runtime none (permanent)
CVE-2026-25679, CVE-2026-27142 Go stdlib CVEs in aws-lambda-rie v1.33 (Go 1.25.7); fixed in Go 1.25.8, pending new RIE release; using releases/latest so will auto-resolve 2026-06-12

ECR scan tooling improvements

  • Surface filePath in failure output (critical for diagnosing vendored copies and false positives)
  • Add inspectorScore, exploitAvailable, epss.score, remediation to failure output
  • Use top-level fixAvailable field to skip genuinely unfixable CVEs (eliminates need to allowlist stdlib/OS CVEs with no available fix)

The following CVEs are now fixed in AL2023 and will be resolved by
the existing dnf upgrade --security in runtime stages:

- CVE-2022-46908 (sqlite, ALAS2023-2025-971)
- CVE-2024-56171 (libxml2, ALAS2023-2025-896)
- CVE-2025-24928 (libxml2, ALAS2023-2025-896)
- CVE-2025-13151 (libtasn1, ALAS2023-2026-1395)
- CVE-2025-13601 (glib2, ALAS2023-2025-1311)
- CVE-2025-1390 (libcap, ALAS2023-2025-897)
- CVE-2025-14087 (glib2, ALAS2023-2025-1349)
- CVE-2025-15467 (openssl, ALAS2023-2026-1406)
- CVE-2025-15468 (openssl, ALAS2023-2026-1434)
- CVE-2025-4802 (glibc, ALAS2023-2025-1001)

Reduces allowlist from 35 to 25 entries.

Signed-off-by: Junpu Fan <junpu@amazon.com>
@aws-deep-learning-containers-ci aws-deep-learning-containers-ci bot added authorized Size:S Determines the size of the PR labels Mar 11, 2026
Signed-off-by: Junpu Fan <junpu@amazon.com>
junpuf added 2 commits March 11, 2026 15:59
The workflow triggered but build-images was skipped because the
check-changes build-change filter didn't include the allowlist path.

Signed-off-by: Junpu Fan <junpu@amazon.com>
The NVIDIA CUDA base image pins to an older AL2023 release version,
so dnf upgrade --security misses patches available in newer releases.
Adding --releasever latest ensures all available security fixes are
applied, including fixes for sqlite, libxml2, libtasn1, glib2, libcap,
openssl, and glibc CVEs.

Signed-off-by: Junpu Fan <junpu@amazon.com>
@junpuf junpuf changed the title fix: remove 10 CVEs from lambda allowlist now patched in AL2023 fix: apply AL2023 security patches to Lambda GPU images Mar 11, 2026
With --releasever latest, dnf upgrade --security now picks up fixes
for libxml2, glib2, expat, libarchive, gnupg2, openssl, and libxslt
CVEs. Reduces allowlist from 25 to 14 entries.

Remaining entries are Python stdlib (tarfile), bundled setuptools/
urllib3, NVIDIA CUDA base image Go tooling, and unpublished CVEs
that cannot be resolved via OS package updates.

Signed-off-by: Junpu Fan <junpu@amazon.com>
Install awslambdaric via uv instead of inheriting it from the Lambda
base image. Copy only Python binary/stdlib/lib from lambda-python stage,
drop site-packages entirely, and install all deps (including
awslambdaric==3.1.1 and pip==25.3) via uv so we own the full dependency
tree. Remove pip after OSS compliance step runs.

This eliminates the three pip-bundled CVEs that could not be patched via
requirements pinning:
- CVE-2024-6345 (setuptools RCE via pip/_vendor/pkg_resources)
- CVE-2025-47273 (setuptools path traversal via pip/_vendor/pkg_resources)
- CVE-2026-21441 (urllib3 decompression bomb via pip/_vendor/urllib3==1.26.20)

Allowlist reduced from 14 to 11 entries.

Signed-off-by: Junpu Fan <junpu@amazon.com>
@aws-deep-learning-containers-ci aws-deep-learning-containers-ci bot added the Size:XL Determines the size of the PR label Mar 12, 2026
junpuf added 2 commits March 12, 2026 11:14
pip is needed only by setup_oss_compliance.sh during the build.
Install it via uv in each builder stage rather than pinning it in
requirements files, so it never appears as an explicit dependency.

Signed-off-by: Junpu Fan <junpu@amazon.com>
setup_oss_compliance.sh calls python -m pip at build time. Since we no
longer inherit site-packages from the Lambda base image, pip must be
explicitly installed via requirements.

Signed-off-by: Junpu Fan <junpu@amazon.com>
@junpuf junpuf force-pushed the patch-lambda-os-cves branch from 3685312 to a6889ff Compare March 12, 2026 18:16
junpuf added 8 commits March 12, 2026 11:34
ECR scan detects setuptools 38.5.0 inside pip/_vendor/pkg_resources
bundled within ensurepip/_bundled/pip-25.3-py3-none-any.whl.
The bundled wheel is only used to bootstrap pip into new virtualenvs,
which never happens in a Lambda runtime. Remove it.

Signed-off-by: Junpu Fan <junpu@amazon.com>
…positives

ECR scanner attributes setuptools 38.5.0 to pip_licenses-5.5.1.dist-info/METADATA
but no such version exists in the runtime image. setuptools==78.1.1 is installed
and both pip and ensurepip/_bundled (which contained the old vendored copy) have
been removed from the runtime image.

Signed-off-by: Junpu Fan <junpu@amazon.com>
deep_learning_container.py imports botocore.session for IMDS calls.
Previously provided by Lambda base image bundled site-packages;
now that we install site-packages ourselves, botocore must be explicit.
cupy and pytorch requirements already include boto3 (which pulls botocore).

Signed-off-by: Junpu Fan <junpu@amazon.com>
Consistent with cupy and pytorch requirements which both pin boto3==1.40.4.

Signed-off-by: Junpu Fan <junpu@amazon.com>
setuptools 82.0.1 vendors jaraco.context 6.1.0 which fixes CVE-2026-23949.
Remove CVE-2026-23949 from allowlist.

Signed-off-by: Junpu Fan <junpu@amazon.com>
…dates

All 4 Go CVEs (CVE-2025-47912, CVE-2025-58188, CVE-2026-25679, CVE-2026-27142)
are in aws-lambda-rie v1.33 built with Go 1.25.7. Fixed in Go 1.25.8 (2026-03-05)
but no new RIE release yet. We use releases/latest so will auto-resolve.
Extended review_by to 2026-06-12.

Signed-off-by: Junpu Fan <junpu@amazon.com>
vulnerablePackages entries include filePath indicating exactly where the
scanner found the vulnerable package. Include it in the error log to make
false positives and vendored copies immediately diagnosable.

Signed-off-by: Junpu Fan <junpu@amazon.com>
…s, remediation

- Use top-level fixAvailable field to skip unfixable CVEs (more reliable than
  per-package fixedInVersion heuristic)
- Add inspectorScore, exploitAvailable, epss.score to failure header
- Add per-package remediation command (e.g. 'sudo dnf check-update')
- Retain filePath from previous commit

Signed-off-by: Junpu Fan <junpu@amazon.com>
@junpuf junpuf changed the title fix: apply AL2023 security patches to Lambda GPU images fix: Lambda GPU image security hardening Mar 12, 2026
junpuf added 6 commits March 12, 2026 16:16
Signed-off-by: Junpu Fan <junpu@amazon.com>
- CVE-2024-6345, CVE-2025-47273: ECR scanner false positives on pip_licenses dist-info
- CVE-2026-25679, CVE-2026-27142: Go CVEs in aws-lambda-rie v1.33 (Go 1.25.7),
  fixed in Go 1.25.8, pending new RIE release

All tarfile stdlib CVEs and GnuPG CVE-2026-24882 dropped from allowlist —
fixAvailable=NO so the scan filter now correctly ignores them.

Signed-off-by: Junpu Fan <junpu@amazon.com>
Remove pip deletion from all 3 runtime stages — customers building on top
of these images need pip to install additional packages.

Keep ensurepip/_bundled removal (bootstrap artifact, not needed at runtime).

Update allowlist reason for CVE-2024-6345 and CVE-2025-47273: setuptools 38.5.0
is vendored inside pip/_vendor/pkg_resources but is never invoked at Lambda
runtime; only the installed setuptools==82.0.1 is used.

Signed-off-by: Junpu Fan <junpu@amazon.com>
Signed-off-by: Junpu Fan <junpu@amazon.com>
CVE-2024-6345 and CVE-2025-47273 are confirmed ECR scanner false positives —
scanner attributes setuptools 38.5.0 to pip_licenses dist-info/METADATA
regardless of whether pip is present or absent in the image.

Signed-off-by: Junpu Fan <junpu@amazon.com>
Latest pip release. Note: pip still vendors urllib3==1.26.20 internally
(pip/_vendor/urllib3) — this is a known upstream issue; pip has been stuck
on urllib3 1.x for compatibility reasons regardless of version.

Signed-off-by: Junpu Fan <junpu@amazon.com>
junpuf added 2 commits March 12, 2026 17:27
These CVEs are permanently unfixable as long as pip is present in the image
(pip vendors old setuptools/urllib3 by design). No review date needed.

Signed-off-by: Junpu Fan <junpu@amazon.com>
@junpuf junpuf enabled auto-merge (squash) March 13, 2026 00:46
@junpuf junpuf merged commit 175db42 into main Mar 13, 2026
25 checks passed
@zhuofuAMZ zhuofuAMZ deleted the patch-lambda-os-cves branch March 13, 2026 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

authorized Size:L Size:M Size:S Determines the size of the PR Size:XL Determines the size of the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants